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SUMMARY  AND  CONCLUSIONS 


Problem 


The  advent  of  a  more  streamlined  Navy  operating  under  reduced 
manning  levels  and  heightened  operational  requirements  imposes  the 
need  for  accurate  human-performance  evaluation  of  ship  personnel  systems. 

On  the  personnel  systems  level,  electronic  technician  reliability  measure¬ 
ment  is  a  necessary  and  integral  part  in  the  evaluation  of  particular 
combat  systems  of  which  technicians  are  components.  The  objective  is 
then  to  develop  and  evaluate  human-performance  reliability  estimates  so 
as  to  be  able  to  effectively  alter  the  personnel  system  in  order  to  max¬ 
imize  the  overall  performance  of  the  system.  The  purpose  of  this  project 
is  to  validate  the  utility  and  effectiveness  of  a  unique  human  perform¬ 
ance  measurement  procedure  developed  under  a  prior  Office  of  Naval  Research 
project  and  designed  to  improve  upon  existing  performance  measuring  tech¬ 
niques  in  a  systems  environment. 


Background  and  Requirements 

Human  reliability  performance  estimation  can  be  accomplished  by 
considering  the  individuals  being  evaluated  as  components  of  a  personnel 
system.  This  consideration  allows  the  use  of  much  of  the  theory  al¬ 
ready  applied  to  equipment  reliability  estimation  to  be  adopted  to 
human-performance  estimation.  After  this  theory  is  applied  to  evalu¬ 
ate  the  performance  of  human  components  in  a  personnel  system,  appro¬ 
priate  combinations  of  the  individual  performance  estimators  will  provide 
a  performance  or  reliability  estimate  of  the  personnel  system  itself. 

In  order  to  improve  upon  existing  performance  estimators  of  the 
human  component  in  a  personnel  system,  Dr.  Arthur  I.  Siegel  and  his 
associates  of  Applied  Psychological  Services,  Inc.,  Wayne,  Pennsyl¬ 
vania,  developed  fleet  post-training  performance  criteria  for  electronic 
maintenance  personnel  with  the  support  of  the  Office  of  Naval  Research. 
The  cumulation  of  these  efforts  resulted  in  the  development  of  unique 
human  performance  measurement  techniques,  closely  allied  with  equipment 
reliability  estimation  techniques.  Siegel  also  developed  procedures  for 
combining  the  technician  performance  estimates  in  appropriate  ways  in 
order  to  estimate  team,  ship  or  squadron  performance. 

An  outgrowth  of  the  prior  research  effort  was  the  suggestion  that 
the  techniques  be  introduced  on  a  limited  basis  to  determine  how  they 
may  be  modified  or  elaborated  upon.  The  Naval  Personnel  Research  and 
Development  Laboratory  is  presently  validating  the  utility  and  effec¬ 
tiveness  of  these  techniques.  The  main  technical  objectives  of  this 
validation  effort  are  to  determine  the  validity  of  the  performance 
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measurement  techniques,  identify  the  restrictions  or  modifications 
required  in  order  to  maximize  their  validity,  and  to  comment  on  the 
statistical  properties  of  those  techniques  as  related  to  their  effec¬ 
tiveness  in  an  operational  context. 

Approach 

In  order  to  realize  an  efficient  and  timely  data  collection  effort 
optical  scanning  instruments  were  utilized  similar  to  those  employed  by 
Applied  Psychological  Services  in  prior  research  efforts. 

The  main  data  collection  instruments  were: 

1.  Personnel  Identification  Information  Forms  (PIIF)  -  this 
form  records  demographic  data  on  the  technician  being  evalu¬ 
ated  . 

2.  Technical  Proficiency  Checkout  Form  (TPCF)  -  this  form 
records  the  level  of  technical  complexity  at  which  a  man 
is  able  to  perform  without  direct  supervision. 

3.  Job  Performance  Questionnaire  (JPQ)  ANSWER  SHEET  -  this 
form  records  supervisory  estimates  of  the  total  number  of 

a  technician's  uncommonly  (EUE)  and  ineffective  performances 
(EUI)  that  the  supervisor  has  observed  during  a  specified 
time  period. 


On  each  of  the  above  instruments  an  individual  in  one  of  the  elec¬ 
tronic  maintenance  ratings  EM,  ET,  FT,  IC,  RD,  RM,  ST,  and  TM  was  evalu 
ated  by  his  supervisor.  On  the  basis  of  the  total  number  of  uncommonly 
effective  (TUE)  and  the  total  number  of  uncommonly  ineffective  (IUI)  in 
cidents  of  performance  recorded  on  the  Job  Performance  Questionnaire 
(JPQ),  four  different  performance  estimators  were  developed.  These 
estimators  are  functions  of  the  total  number  of  uncommonly  effective 
(EUE)  and  the  total  number  of  uncommonly  ineffective  (EUI)  incidents  of 
performance  observed  by  the  supervisor  on  each  of  eight  job  dimensions 
characteristic  of  electronic  maintenance  activities.  The  four  estima¬ 
tors  of  human  reliability  are: 

1.  Series  Reliability  Estimate  (SRE) 

2.  Series-Parallel  Reliability  Estimate  (PRE) 

3.  Geometric  Mean  Reliability  Estimate  (GRE) 

4.  Weighted-Average  Reliability  Estimate  (WRE) 
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Utilizing  the  Technical  Proficiency  Checkout  Form  as  a  criterion 
measure,  the  validity  of  each  of  the  four  estimators  was  established. 
Finally  an  appropriate  statistical  analysis  of  the  performance  data 
revealed  areas  of  difficulty  with  each  of  the  four  estimators  and  sug¬ 
gested  modifications  required  to  increase  their  efficiency. 


Findings 

The  relevant  findings  of  this  preliminary  investigation  of  the 
sample  data  resulted  mainly  from  an  analysis  of  the  distributional  pro¬ 
perties  of  the  predictor  and  criterion  variables.  From  an  application 
of  Geary's  test  for  normality  it  was  found  that  only  the  predictor 
variable  WRE  could  be  termed  normally  distributed.  As  a  result  of  this 
finding,  a  curvilinear  multiple  regression  technique  was  applied  to  a 
portion  of  the  sample  data  with  an  emphasis  upon  the  least-squares 
analysis  resulting  from  an  application  of  this  technique. 

A  comparison  of  the  multiple  correlation  coefficients  resulting 
from  the  curvilinear  regression  analysis  revealed  that  in  every  case 
a  straight-line  was  the  best  fit  to  the  performance  data.  The  product- 
moment  correlation  between  the  predictor  variables  (SRE,  PRE,  GRE,  and 
WRE)  and  criterion  variable  were  respectively  -.069,  -.009,  .024  and 
.242  indicating  moderate  validity  on  the  part  of  the  WRE  for  appraising 
the  absolute  level  of  technician  performance. 


Conclusions 


Even  though  the  statistical  analysis  for  this  report  was  con¬ 
cerned  with  the  data  collected  at  only  one  location  (Flotilla  NINE, 

San  Diego,  California),  tentative  conclusions  can  be  made  on  the  char¬ 
acteristics  of  the  performance  evaluators. 

The  extreme  skewness  of  the  distributions  of  the  SRE,  PRE,  and 
GRE  will  probably  not  be  possible  to  avoid,  thus  eliminating  many  con¬ 
venient  analyses  of  the  data.  In  particular  the  distribution  of  the 
criterion  variable  as  derived  from  the  TPCF  is  so  markedly  skewed  that 
no  improvement  in  it  is  likely  with  larger  sample  sizes.  The  moderate 
validity  of  the  WRE  indicates  that  the  technique  of  deriving  individual 
performance  estimates  as  a  function  of  supervisory  estimates  of  uncom¬ 
monly  effective  and  uncommonly  ineffective  performance  is  a  promising 
area  but  requiring  further  research.  Although  the  other  performance 
estimators  did  not  fare  as  well  as  the  WRE,  it  is  clear  that  they  could 
be  improved  upon  if  certain  areas  of  difficulty  are  avoided.  It  can 
be  concluded  that  none  of  these  areas  of  difficulty  are  insurmountable. 
In  fact  they  have  revealed  the  appropriate  modifications  that  are  needed 
in  order  to  increase  the  validity  of  the  performance  estimators,  e.g., 
by  evaluating  only  those  individuals  who  actually  do  work  at  a  high 
proportion  of  the  job  activities. 
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Research  is  continuing  upon  the  evaluation  of  the  performance 
measurement  technique  with  respect  to  the  totality  of  data  collected. 

It  is  felt  that  the  larger  data  base  will  reveal  other  aspects  of  the 
performance  measurement  technique,  particularly  when  individual  ratings 
are  being  considered. 
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INTRODUCTION 


The  purpose  of  this  report  is  to  comment  on  the  data  collection 
effort  and  initial  data  reduction  methods  and  analyses  that  have  been 
performed  to  date  for  the  ONR  work  unit  entitled  Personnel  Technology: 
Relating  Individual  Performance  Effectiveness  to  Unit  and  Ship  Effec¬ 
tiveness  (Office  of  Naval  Research,  Project  Order  Number  2-0046,  NR 
150-336).  The  goal  of  this  research  project  is  to  provide  an  empirical 
basis  for  assessing  the  utility  to  the  Navy  of  a  performance  measure¬ 
ment  technique  developed  under  a  prior  ONR  contract.  Under  that  contract 
Dr.  Arthur  I.  Siegel,  Philip  J.  Federman,  and  their  associates  of  Applied 
Psychological  Services,  Inc.,  Wayne,  Penn.,  developed  fleet  post-training 
performance  evaluative  measures  which  seemed  to  be  of  value  for  eventual 
widespread  implementation  within  the  report  -  Development  of  Performance 
Evaluative  Measures:  Investigation  into  and  Application  of  a  Fleet 
Post-Training  Performance  Evaluative  System  [12],  Furthermore,  the 
research  effort  undertaken  by  Naval  Personnel  Research  and  Development 
Laboratory  (NAVPERSRANDLAB)  is  not  only  to  replicate  the  efforts  of 
Siegel  and  Federman  [12]^,  but  also  to  further  research  their  techniques 
and  similarly  related  performance  measurement  techniques. 


BACKGROUND 


In  order  to  better  appreciate  the  techniques  employed  in  this  report, 
it  will  be  necessary  to  discuss  the  development  of  the  performance  measure¬ 
ment  techniques  which  were  employed  by  Siegel  and  Federman  [12],  Funda¬ 
mentally  the  authors  had  employed  magnitude  estimates  of  functions  of 
critical  incidents.  In  the  past  this  has  seemed  to  be  a  valid  approach 
to  obtain  estimates  of  human  performance. 

Generally  the  main  problem  is  to  estimate  the  performance  of  a  par¬ 
ticular  personnel  system  as  a  function  of  the  performance  of  individuals 
that  are  a  part  of  the  system.  This  necessarily  reduces  personnel  system 
performance  estimation  to  a  discussion  of  estimators  of  individual  per¬ 
formance  where  individuals  are  the  components  of  the  system.  Compounding 


■^All  numbers  enclosed  in  brackets  refer  to  corresponding  numbers 
of  documents  and  publications  listed  under  References. 
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the  individual  estimates  in  a  meaningful  way  will  provide  estimates  of 
personnel  system  performance. 

Critical  Incidents  Techniques  in  Personnel  Performance  Estimation 

Let  EUE  (EUI)  be  the  total  number  of  uncommonly  effective  (uncommonly 
ineffective)  incidents  of  performance  observed  by  a  rater  in  a  certain 
time  period  on  some  individual  under  observation.  Using  these  functions 
of  critical  incidents,  previous  researchers  (see,  for  example,  Whitlock 
[17],  et  al.)  demonstrated  that  there  is  a  definite  straight  line  or 
curvilinear  relationship  between  EUE  (or  the  ratio  EUE/EUI)  and  correspond¬ 
ing  performance  evaluations.  Prior  results  such  as  these  seemed  to  leave 
little  doubt  that  a  critical  incidents  technique  to  performance  evaluation 
is  a  valid  and  useful  approach. 

Following  upon  the  results  of  Whitlock,  et  al.,  Siegel  has  further 
developed  and  applied  the  above  mentioned  techniques  to  the  post- training 
performance  evaluation  of  individuals  in  various  avionic  or  electronic 
ratings  in  the  U.  S.  Navy.  In  particular  Siegel  and  Pfieffer  [14]  sug¬ 
gested  that  judgments  of  uncommonly  effective  and  uncommonly  ineffective 
performances  possess  merit  as  useful  indicators  of  overall  personnel  pro¬ 
ficiency.  The  researchers  employed  magnitude  estimates  of  the  number  of 
uncommonly  effective  and  uncommonly  ineffective  performances  relative  to 
a  short  prior  period  for  avionic  personnel.  They  derived  a  performance 
index  from  the  ratio  of  the  sum  of  uncommonly  effective  (EUE)  performances 
to  the  sum  of  uncommonly  effective  plus  the  sum  of  uncommonly  ineffective 
(EUI)  performances,  namely  (EUE/ [EUE  +  EUI]).  Siegel  and  Pfeiffer  [14] 
concluded  that:  (1)  magnitude  estimates  of  uncommonly  effective  and  in¬ 
effective  performances  yielded  useful  data  which  could  form  the  basis 
for  a  personnel  subsystem  reliability  index;  (2)  the  ratio  of  the  sum  of 
uncommonly  effective  plus  the  sum  of  uncommonly  ineffective  performances 
yields  an  index  which  discriminates  in  the  anticipated  direction;  and, 

(3)  the  obtained  avionic  personnel  subsystem  index  could  be  utilized  for 
post-training  performance  appraisal,  personnel  placement,  and  squadron 
evaluative  purposes. 

The  Job  Performance  Questionnaire  as  an  Instrument  for  Obtaining  Estimates 
of  EUE  and  EUI 


The  Job  Performance  Questionnaire  (JPQ)  is  an  instrument  for  providing 
information  on  supervisory  estimates  of  EUE  and  EUI.  Siegel  and  Federman 
[12]  demonstrated  the  utility  and  practicality  of  a  Job  Performance 
Questionnaire  (see  Appendix  A  of  this  paper)  for  technicians  in  the  eight 
electronic  maintenance  ratings  (EM,  ET,  FT,  IC,  RD,  RM,  ST,  and  TM) .  From 
an  evaluation  of  499  technicians  in  those  ratings,  the  researchers  found 
that  the  JPQ  yields  on  estimate  of  the  total  number  of  uncommonly  effective 
and  uncommonly  ineffective  incidents  of  behavior  on  eight  job  activity 
factors.  The  eight  job  factors  and  their  definitions  are  given  in 
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Appendix  B.  The  job  activity  factors  were  isolated  by  Siegel  and  Schult2 
[15]  and  were  descriptive  of  naval  avionic  electronic  maintenance  jobs. 

It  was  the  contention  of  Siegel  and  Federman  [12]  that  estimates  of 
uncommonly  effective  and  of  uncommonly  ineffective  behavior  along  eight 
dimensions  of  job  activities  could  be  compounded  into  a  meaningful  measure 
of  technician  effectiveness.  Moreover,  the  individual  technician  ef¬ 
fectiveness  values  can  be  further  treated  to  form  effectiveness  values 
for  ratings,  ships,  and  squadrons.  In  particular,  for  each  job  activity 
the  reliability  ratio  (£UE/[EUE  +  EUI]),  as  employed  in  the  prior  reports 
[12,  14],  yields  an  estimate  of  the  probability  of  effective  performance 
for  the  technician  and  job  activity  considered.  The  reliability  ratios  are 
then  compounded  in  meaningful  ways  to  provide  estimates  of  individual 
effectiveness  or  reliability  of  on-the-job  performance. 

Various  Procedures  for  Compounding  Reliability  Ratios  to  Estimate  Technician 
Reliability 

Employing  the  reliability  ratios  defined  above,  Siegel  and  Federman 
[12]  have  employed  those  ratios  to  develop  the  following  reliability 
estimates: 

1)  SERIES  RELIABILITY  ESTIMATE  (SRE) 

The  series  reliability  measure  of  total  effectiveness 
for  an  individual  is  derived  by  multiplying  individual 
job  activity  reliability  ratios  to  yield  a  total  reli¬ 
ability  score,  i.e., 

Rs  ■  rl  x  r2  x  • • •  x  r8 
where  Rg  =  series  reliability^,  and 

r.  =  (EUE/[ZUE  +  EUI])  is  the  reliability  ratio  for 

1  . th  .  ,  .  . 

l  job  activity. 

It  is  to  be  noted  that  use  of  the  series  reliability 
estimate  requires  the  assumption  that  performance  reli¬ 
ability  on  each  job  activity  is  independent  of  perform¬ 
ance  on  other  job  activities. 


2 

It  is  helpful  to  note  that  the  series  reliability  estimate 
possesses  the  following  properties: 

a)  for  each  of  the  i  =  1,  ...,  8  job  activities 
0  _<  r  £  1,  and,  therefore, 

b)  0  <  R  <1 

—  s  — 

c)  R  <  smallest  r.. 

s  —  1 
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2)  SERIES-PARALLEL  RELIABILITY  ESTIMATE  (PRE) 

It  has  been  stated  by  Siegel  and  Federman  [12]  that 
"the  series  and  series-parallel  reliabilities  provide 
measures  of  personnel  proficiency  relative  to  perform¬ 
ance  on  the  entire  job  (i.e.,  all  eight  job  activities)." 
The  series-parallel  estimate  of  individual  proficiency  is 
defined  as: 


R  =  R  x(2  -  r.  )  x  ...  x(2  -  rQ) 
p  s  1  o 

where  R  ,  r.(i  =  1,  ....  8)  are  defined  in  1)  above, 
s  i  ’  ’ 

This  particular  estimate  tends  to  provide  a  more 
optimistic  estimate  of  individual  performance,  however, 
its  content  validity  and  derivation  is  deserving  of  further 
development. 

3)  GEOMETRIC  MEAN  RELIABILITY  ESTIMATE  (GRE) 

Let  r^,  r*,  r*  and  r^  be  the  four  highest  job 
activity  reliability  ratios  of  the  eight  reliability 
ratios  for  a  technician  being  evaluated.  The  geometric 
mean  reliability  for  the  technician  is  defined  as: 


This  particular  estimate  is  an  estimate  of  individual 
performance  that  stresses  the  strong  points  of  an  individuals 
performance.  However,  it  also  tends  to  ignore  his  weak 
points  and  therefore  should  be  used  with  caution. 

Development  of  a  Criterion  for  Validating  Performance  Estimators 

In  order  to  determine  the  validity  of  the  above  estimates  (SRE, 
PRE,  and  GRE)  in  estimating  the  absolute  level  of  an  individual’s 
performance,  performance  data  were  collected  by  Siegel  and  Federman 
[12]  by  means  of  an  evaluative  instrument  called  the  Technical  Pro¬ 
ficiency  Checkout  Form  (TPCF)  (see  Appendix  C  of  this  paper).  The 
TPCF  consists  of  eight  job  tasks  listed  in  hierarchial  order  from 
easiest  to  most  difficult.  The  eight  tasks  meet  the  Guttman  re¬ 
quirements  for  scalability  (see,  for  example,  Guttman  [9]).  Siegel, 
Schultz,  and  Lanterman  [16],  1964,  employed  the  scale  underlying  the 
eight  tasks  to  determine  the  cutting  points  for  placing  avionic  petty 
officers,  third  class  and  strikers  in  one  of  the  three  levels  of 
technical  proficiency.  The  three  TECHNICAL  PROFICIENCY  CHECKOUT  (TPC) 


levels  are: 

Level 

1: 

above  desirable 

Level 

2: 

below  desirable  but  at  least  minimally  acceptable 

Level 

3: 

below  minimally  acceptable. 
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The  trichotomous  division  was  based  on  supervisor’s  judgments  of  the 
performance  level  required  for  achieving  the  objectives  given  in  Ap¬ 
pendix  D  of  this  paper. 


In  order  to  facilitate  understanding  of  the  procedure  for  plac 
ing  an  individual  in  one  of  the  TPC  levels,  define  the  function 
F  (i  =  1,  . . . ,  8)  as : 

til 

f  1  if  the  technician  is  checked  out  on  the  i 


task  of  the  TPCF 

0  if  the  technician  is  not  checked  out  on  the 
ifch  task  of  the  TPCF. 


The  TECHNICAL  PROFICIENCY  (TP)  score  for  a  technician  is  then  defined  as 

f  F..  Finally,  in  the  prior  report  [16],  the  procedure  for  determining 
i=l  1 

the  TPC  level  is  given  by: 

£ 

a)  add  0.5  to  TP  score  for  an  individual.  Let  TP  be  the 
resultant  score. 

b)  if  TP*  <  3.92,  then  TPC  level  =  3 

c)  if  3.92  £  TP*  <_  5.63,  then  TPC  level  =  2 

d)  if  TP*  >  5.63,  then  TPC  level  =  1. 

In  a  later  study  Siegel  and  Fischl  [13]  correlated  technicians  TPC 
levels  with  the  technicians  total  scores  on  a  performance  test.  Employing 
a  triserial  correlation  coefficient  (see,  for  example,  Jaspen  [10])  as 
an  estimate  of  the  product-moment  correlation,  they  found  a  triserial 
correlation  of  .40.  When  corrected  for  the  lack  of  perfect  reliability 
in  the  performance  test  criterion,  its  value  became  .74.  On  the  basis 
of  their  investigation  of  the  concurrent  validity  of  the  TPCF,  they  con¬ 
cluded  that  "  the  Technical  Proficiency  Checkout  Form,  previously  shown 
to  be  reliable  and  practical,  piay  now  be  considered  to  possess  a  subs¬ 
tantial  degree  of  validity  for  appraising  the  absolute  proficiency  level 
of  avionics  technicians  in  the  fleet.”  Finally,  Siegel  and  Federman 
[12]  recorded  a  triserial  correlation  of  .38  between  the  TPC  level  of  the 
technicians  evaluated  and  their  Series  Reliability  Estimate  (SRE) , 
concluding  there  is  some  basis  to  believe  that  the  SRE  correlates  with 
on-the-job  performance. 

Main  Results  of  Prior  Studies 

While  there  are  other  results  which  many  of  the  previous  studies 
have  found  relative  to  the  post-training  performance  estimation  of 
technicians  involved  in  fleet  electronic  maintenance  activities,  this 
report  has  been  and  will  only  be  concerned  with  the  validity  of  the 
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Series  Reliability  Estimate  (SRE) ,  Series-Parallel  Reliability  Estimate 
(PRE) ,  Geometric  Mean  Reliability  Estimate  (GRE)  in  estimating  on-the- 
job  performance  where  the  TPCF  is  the  primary  measure  or  criterion  of 
on-the-job  performance.  Some  of  the  more  important  conclusions  of  prior 
reports  relative  to  content  validity  of  the  SRE,  PRE,  and  GRE  are: 

1.  Reliability  ratios  of  the  form  EUE/ (ZUE  +  ZUI)  indicate 
the  probability  of  effective  performance  on  a  particular 
job  activity  for  the  technician  being  evaluated. 

2.  The  JPQ  is  an  instrument  for  providing  magnitude  estim¬ 
ates  of  EUE  and  ZUI  for  each  man  being  evaluated  by  his  immed- 
ate  supervisor. 

3.  The  TPCF  possesses  a  substantial  degree  of  validity  for 
appraising  the  absolute  level  of  technician  proficiency  and 
may  be  employed  as  a  criterion  for  the  JPQ. 

4.  There  is  some  basis  (triserial  correlation  of  .38  with 
TPC  level)  to  believe  that  the  SRE  is  a  good  estimator  of 
on-the-job  performance. 


THE  DATA  COLLECTION  EFFORT 


The  data  collection  effort  that  this  report  will  be  concerned  with 
was  conducted  with  ships  from  Commander,  Cruiser-Destroyer  Flotilla 
NINE,  Pacific  Fleet,  during  the  months  of  March  and  April,  1972.  The 
methodology  for  the  data  collection  required  a  project  coordinator  at 
Destroyer  Flotilla  NINE  to  seek  the  ships  (all  of  which  were  destroyers) 
to  take  part  in  the  project  and  the  assignment  of  a  liaison  officer 
aboard  each  ship.  Essentially  the  project  coordinator  served  to  coor¬ 
dinate  the  activities  of  the  project  researchers  of  NAVPERSRANDLAB  and 
those  of  the  ship  liaison  officers.  Initially  a  meeting  was  arranged 
between  the  ship  liaison  officers  and  the  project  researchers  by  the 
project  coordinator.  In  the  initial  meeting  the  researchers  acquainted 
the  liaison  officers  of  the  purpose  of  the  project  and  their  subsequent 
duties  in  the  data  collection  effort. 

A  total  of  11  ships  from  Destroyer  Flotilla  NINE  participated  in 
the  project  with  582  technicians  being  evaluated  by  their  immediate 
supervisors.  As  in  the  previous  report  [12],  the  technicians  were 
in  one  of  the  eight  electronic  ratings  EM,  ET,  FT,  IC,  RD,  RM,  ST, 
and  TM. 
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Data  Collection  Instruments 


The  performance  evaluation  forms  that  were  completed  by  each  super¬ 
visor  are  given  in  Appendices  C,  E,  and  F.  In  particular  the  forms  were: 

a)  Job  Performance  Questionnaire  (JPQ)  ANSWER  SHEET 
This  form,  an  example  of  which  is  in  Appendix  E, 

serves  the  same  purpose  as  the  JPQ  of  the  previous 
report  [12]  given  in  Appendix  A  of  this  paper,  i.e., 
to  record  estimates  of  the  total  number  of  uncommonly 
effective  (EUE)  and  uncommonly  ineffective  (IUI) 
performances  the  supervisor  has  observed  on  the  man 
he  is  evaluating. 

b)  Technical  Proficiency  Checkout  Form  (TPCF) 

This  form  is  essentially  identical  to  the  TPCF  used 
by  Siegel  and  Federman  [12]  and  was  discussed  in  detail 
in  the  Bachground  section  of  this  report. 

c)  Personnel  Identification  Information  Form  (PIIF) 

This  form,  an  example  of  which  is  in  Appendix  F, 

was  concerned  with  the  background  data  of  the  man 
being  evaluated.  It  was  completed  in  part  by  his 
supervisor  with  the  administrative  officer  providing 
the  remaining  information. 


Results  of  the  Data  Collection  Effort 


All  forms  were  returned  in  a  useable  condition  with  very  few  errors 
in  completion  and  little  missing  data.  All  of  the  missing  data  was  con¬ 
cerned  with  background  information  as  recorded  on  the  PIIF  (Appendix  F) 
and  in  no  way  influences  the  results  of  this  report.  Finally  the 
PIIF  and  JPQ  ANSWER  SHEET,  being  optical  scanning  sheets,  were  reduced 
to  computer  cards  ready  for  the  subsequent  data  analysis. 

The  entire  data  collection  effort  briefly  discussed  above  follows 
essentially  the  procedure  employed  in  the  previous  report  [12].  Every 
effort  has  been  made  to  insure  accuracy  and  correctness  of  the  data 
analysis  to  be  discussed  in  the  next  section.  Also,  to  achieve  a 
similarity  in  results,  every  effort  has  been  made  to  perform  a  statis¬ 
tical  analysis  identical  to  that  of  the  previous  report  [12].  Where 
this  is  not  possible  or  appropriate,  the  changes  or  deviations  in 
analysis  from  that  of  the  previous  report  [12]  will  be  commented  on. 
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DATA  ANALYSIS  AND  OBSERVATIONS 


In  order  to  initiate  the  data  analysis  it  will  be  necessary  to  in¬ 
vestigate  in  some  detail  a  fundamental  problem  relating  to  the  Series, 
Series-Parallel,  and  Geometric  Mean  Reliability  estimates  (the  SRE, 

PRE,  and  GRE  respectively). 


Cases  of  Difficulty  in  Calculating  Performance  Estimates 

The  reader  will  recall,  as  discussed  in  the  Background  section  of 
this  report  that  reliability  ratios  of  the  form  (EUE/[ZUE  +  ZUI])  were 
derived  for  each  man  on  each  of  eight  job  activities  and  that  these 
ratios  were  combined  in  some  way  to  form  the  SRE,  PRE,  and  GRE.  How¬ 
ever,  the  following  two  cases  require  the  adoption  of  some  convention  in 
order  to  calculate  the  reliability  ratios : 

1)  the  technician  did  not  work  at  that  job  activity,  or 

2)  the  technician  received  EUE  =  0  and  EUI  =  0  by  the 
supervisor,  implying  that  the  reliability  ratio 

q  ~  is  undefined. 

By  observing  the  frequency  with  which  such  cases  occur  across  all  11 
ships,  one  can  determine  the  extent  to  which  any  convention  for  esti¬ 
mating  performance  in  those  cases  would  effect  individual  SRE,  PRE,  and 
GRE  values.  A  complete  discussion  of  this  effect  is  given  in  Appendix 
G  and  the  interested  reader  is  referred  to  that  section  for  a  more 
detailed  account.  For  now  it  suffices  to  say  that  the  above  two  cases 
can  have  a  dramatic  effect  upon  the  individual  performance  estimates 
and  that  these  estimates  will  be  greatly  influenced  by  the  convention 
that  is  adopted. 


The  Adoption  of  a  Convention  for  Estimating  Performance  in  Certain  Job 
Activities 


Siegel  and  Federman  [12]  chose  to  employ  "the  average  value  for 
his  rating  on  his  ship,"  on  those  job  activities  which  the  technician 
did  not  work  at  or  received  EUE  =  0  and  EUI  =  0  by  his  supervisor. 
Unfortunately  the  results  of  the  data  collection  effort  at  San  Diego 
indicated  that  this  technique  was  not  feasible  for  that  data.  The 
main  reason  for  this  is  that  on  every  ship  there  were  ratings  for  which 
in  some  job  activities  those  two  cases  occured  for  all  men  in  that 
rating.  In  Appendix  I  is  provided  a  detailed  account  of  this  observation 
for  the  interested  reader. 
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In  order  to  overcome  this  problem,  the  convention  adopted  in 
this  report  was  to  employ  a  composite  reliability  value  across  all 
ships  for  each  job  activity  and  rating.  Let  IUE  (i,  j)  be  the 
sum  across  all  ships  of  all  ZUE's  over  all  men  in  the  i^h  rating  and 
jfch  job  activity.  Similarily  the  sum  of  all  nil's  is  calculated; 
denote  this  sum  by  IUI  (i,  j).  The  composite  reliability  score  for  the 
ith  rating  and  jfc^  job  activity  is  defined  as 

R(i,j)  =  ZUE(i,j)/[ZUE(i,j)  +  ZUI(i,j)]. 

This  particular  estimate  of  job  performance  provides  an  "expected"  level 
of  effectiveness  for  a  technician  in  the  i^h  rating  and  jth  job  activity 
(for  ships  at  San  Diego).  Appendix  K  gives  the  resulting  composite 
reliability  values  (R(i,j))  for  each  rating  and  job  activity.  For 
example,  from  Appendix  K,  R(l,3)  is  the  composite  reliability  value  for 
EM's  on  job  activity  number  3  -  Electronic  Circuit  Analysis  -  and  is  given 
by  R(l,3)  =  .8465.  For  definiteness,  we  state  that  for  all  EMfs  who 
have  not  worked  at  job  activity  number  3  or  who  received  EUE  =  0  and 
IUI  =  0  for  that  job  activity,  their  reliability  ratio  for  that  job 
activity  is  given  by  r3  =  R(l,3)  =  .8465.  Similarly  such  a  procedure  is 
employed  on  the  other  ratings  and  job  activities. 

The  composite  reliability  score  is  an  estimate  that  can  always  be 
derived  when  many  ships  are  involved,  however,  in  no  way  does  it  over¬ 
come  the  implications  of  the  results  discussed  in  Appendix  G  and  their 
subsequent  effect  on  the  estimates  SRE,.  PRE,  and  GRE. 


The  Weighted-Average  Reliability  Estimator 

In  addition  to  the  three  performance  estimators  (SRE,  PRE,  and 
GRE)  previously  introduced,  this  report  will  also  discuss  an  estimate 
of  the  form 

NJ 

R  =  I  r.  w./NJ 
w  i=l  x  l 

where,  NJ  =  number  of  job  activities  the  technician  actually  worked  at 
r^  =  the  reliability  ratio  for  the  ifch  job  activity 

w.  =  weight  denoting  the  importance  of  the  ifc^  job  activity  in 
estimating  the  technician’s  overall  performance 

The  estimator  Rw  will  be  called  a  Weighted-Average  Reliability 
Estimate  (WRE)  of  technician  effectiveness.  This  estimator 
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has  the  desirable  property  of  providing  a  performance  estimate  on 
only  the  job  activities  the  technician  actually  worked  at  during  the 
rating  period.  By  eliminating  those  job  activities  the  technician 
did  not  work  at,  the  WRE  yields  an  estimate  of  greater  utility. ^ 
Furthermore,  it  tends  to  give  most  (least)  importance  to  job 
activities  which  would  be  more  (less)  indicative  of  the  technician's 
overall  performance. 


Validity  of  the  Performance  Estimators  as  Determined  by  a  Triserial 
Correlation 


A  triserial  correlation  between  each  of  the  four  estimators  (SRE, 
PRE,  GRE,  and  WRE)  and  Technical  Proficiency  Checkout  (TPC)  level  was 
calculated  to  determine  the  validity  of  each  reliability  estimator. 

The  associated  results  are  given  in  Table  1: 


TABLE  1 

TRISERIAL  CORRELATION  INFORMATION  FOR  TECHNICIANS  AT  SAN  DIEGO 


TPC  LEVEL 

Mean  Reliabilities 

in  Each 

TPC  Level 

SRE 

PRE 

GRE 

WRE 

1 

.3507 

.5968 

.9308 

.6565 

2 

.3312 

.5527 

.9194 

.5662 

3 

.4274 

.6245 

.9225 

.5572 

TRI SERIAL 

CORRELATIONS 

-0.0900 

-0.0149 

0.0325 

.2555 

Included  within  the  NJ  job  activities  may  be  cases  where  EUE  =  0 
and  EUI  =  0.  For  those  cases  the  composite  reliability  values  (Appen¬ 
dix  K)  must  be  used  for  the  reliability  ratios.  This  procedure  was 
previously  discussed  on  page  9  . 

2 

Appendix  L  provides  a  detailed  account  of  the  greater  utility  of 
the  WRE  and  an  explanation  of  how  the  weights  (’w. ,  i=l, . .  . ,  NJ)  are 
derived. 
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Because  the  triserial  correlation  coefficient  is  an  estimate  of  the 
product-moment  correlation,  a  test  of  the  hypothesis  that  the  triserial 
correlation  coefficient  for  each  estimate  is  equal  to  zero  is  possible 
(see,  for  example,  Guilford  [8]).  However,  for  only  the  SRE  and  WRE 
could  this  hypothesis  be  rejected  (at  the  a  =  .05  level  of  significance), 
implying  that  the  PRE  and  GRE  are  not  good  linear  estimators  of  the 
absolute  level  of  technician  performance.  On  the  other  hand,  the  triserial 
correlation  of  SRE  and  TPC  level  (-.09)  is  significantly  different  from 
zero  (a  =  .05),  but  the  sign  is  not  in  the  hoped  for  positive  direction. 

The  resulting  triserial  correlation  of  TPC  level  with  WRE  was  .2555, 
which  was  significantly  different  from  zero  (a  =  .05)  and  in  the  right 
direction.  As  such,  at  this  stage  of  the  analysis,  it  must  be  concluded 
that  the  WRE  is  the  superior  type  of  estimator,  even  though,  in  some 
sense,  SRE  possesses  minimal  merit  as  an  estimator. 


The  Appropriateness  of  a  Triserial  Correlation  for  Associating  Performance 
Estimators  with  the  TPCF 


The  previous  analysis  represents  essentially  the  data  analysis 
techniques  adopted  by  Siegel  and  Federman  [12]  but  now  applied  to  the 
sample  data  at  San  Diego.  However,  in  an  attempt  to  employ  a  data 
analysis  appropriate  to  the  data,  consider  the  following  assumptions 
that  one  is  inherently  making  when  applying  the  triserial  correlation 
(see  again  Jaspen  [10]): 

a)  the  segmented  variable  is  basically  continuous  and 
normally  distributed 

b)  all  the  segments  which  together  would  form  a  whole 
normal  distribution  are  present. 

Our  attention  will  be  focused  on  requirement  a).  Consider  the 
histogram  in  Appendix  N.  This  is  a  histogram  of  all  the  Technical  Pro¬ 
ficiency  (TP)  scores  (defined  ,in  the  Background  section  of  this  report) 
for  technicians  evaluated  at  San  Diego.  Recall  that  this  variable, 
Technical  Proficiency  score  (TP  score),  was  segmented  into  one  of  three 
levels  of  technical  proficiency.  If  in  fact  all  the  required  segments 
are  present,  then  the  histogram  in  Appendix  N  represents  the  entire 
distribution  of  the  segmented  variable,  which  may  be  taken  as  continuous. 
Clearly  this  does  not  seem  to  represent. a  normal  distribution.  For  a 
more  valid  proof  of  this  statement,  Geary's  test  for  normality  [7]  was 


11 


applied  to  the  distribution  of  TP  scores.  Geary's  test  statistic  is 
given  by 


2(E'  -  N'X) 

a  =  .  — - - — .  r 

V NEX2  -  (EX)2 

where  X  represents  an  observation,  X  the  sample  mean,  N  the  sample  size, 

E'  the  sum  of  all  observations  greater  than  X,  and  N'  the  number  of 
observations  greater  than  X.  If  the  null  hypothesis  is  that  the  under¬ 
lying  distribution  is  normal,  then  it  has  been  shown  [2]  that 

(a  -  .7979) 

z  =  - 

.2123 

is  approximately  normal  with  mean  zero  and  variance  one.  In  fact  a 
conservative  test  of  the  hypothesis  that  the  underlying  distribution 
is  normal  (at  the  a  =  .05  level  of  significance)  is  given  by: 

reject  the  null  hypothesis  of  normality  if 
z  is  greater  than  1.96  or  less  than  -1.96. 

When  Geary's  test  statistic  was  applied  to  the  sample  data  of  TP 
scores,  the  resulting  test  statistic  values  were 

a  =  .8476,  implying  z  =  5.6429 

Therefore,  the  assumption  of  normality  for  TP  scores  must  be  rejected 
and  so  an  application  of  the  triserial  correlation  is  inappropriate  for 
the  sample  data  collected  at  San  Diego. 

For  future  reference,  when  Geary's  test  for  normality  was  applied  on 
the  distribution  of  SRE,  PRE,  GRE,  and  WRE,  the  following  table  was  derived: 


This  particular  goodness  of  fit  test  has  several  advantages  over  the 
usually  applied  Kolmogorov-Smirnov  tests  or  the  well-known  Chi-square 
tests  in  that,  in  particular,  the  population  mean  and  standard  deviation 
need  not  be  precisely  known  and  the  test  need  not  be  applied  just  to 
large  samples.  Furthermore  Geary's  test  seems  to  be  more  sensitive  to 
departures  from  normality  than  the  other  two  tests  [3,  4,  or  11]. 
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TABLE  2 


GEARY'S  TEST  STATISTIC  VALUES  FOR  THE  PERFORMANCE  ESTIMATORS 


SRE 

PRE 

GRE 

WRE 

a 

.8747 

.8890 

.5456 

.7876 

z 

8.726 

10.3472 

-28.6657 

-1.1716 

Clearly  the  only  estimator  which  is  indicative  of  being  normally  distri¬ 
buted  is  the  WRE.  The  lack  of  normality  on  the  part  of  the  other  estima¬ 
tors  is  not  necessarily  an  undesirable  feature,  but  it  is  true  that  this 
exercise  does  point  out  yet  one  more  desirable  feature  of  the  WRE,  namely, 
its  normality.  The  reader  is  also  referred  to  the  histograms  of  the 
frequency  of  occurrence  of  the  SRE,  PRE,  and  WRE  values  given  in  Appendices 
0,  P,  and  Q  with  class  intervals  for  the  histograms  presented  in  Appendix 
R. 


Curvilinear  Regression  Analysis  as  an  Alternative  to  Triserial  Correlation 

Essentially  due  to  the  non-normality  of  the  TP  scores,  an  alter¬ 
nate  analysis  is  suggested  in  order  to  determine  the  degree  of  associa¬ 
tion  between  the  predictor  variables  (SRE,  PRE,  GRE,  and  WRE)  and 
criterion  variable  (TP  score).  The  particular  procedure  to  be  employed 
to  achieve  this  end  will  be  a  curvilinear  multiple  regression  procedure. 
The  reader  may  review  this  subject  in  any  of  the  texts  that  elaborate  on 
the  procedure.  In  particular  Draper  and  Smith  [5]  give  an  excellent 
concise  explanation  of  all  the  ramifications  of  this  and  related  obser¬ 
vations  on  regression  in  general.  However,  a  few  remarks  on  this  subject 
for  the  purposes  of  this  report  have  been  provided  in  Appendix  S. 

The  curvilinear  regression  analysis  that  will  follow  is  essentially 
a  replication  of  the  technique  in  Cooley  and  Lohnes  [1],  Throughout 
this  analysis  the  predictor  variable  is  one  of  SRE,  PRE,  GRE,  or  WRE 
while  the  criterion  variable  is  the  TP  score.  The  computer  program 
employed  in  [1]  for  curvilinear  regression  is  also  used  in  this  paper. 


13 


Appendices  T,  U,  V,  and  W  of  this  paper  provide  computer  printouts 
as  are  found  in  [1]  but  for  the  predictor  and  criterion  variables  of 
this  report.  The  computer  printouts  are  in  terms  of  "centered"  data 
making  maximum  use  of  the  correlation  matrix.  This  technique  improves 
the  computation  of  the  printout  values  by  minimizing  roundoff  errors. 
Table  3  provides  the  essential  information  for  the  curvilinear  regres¬ 
sion  analysis  and  is  taken  from  the  computer  printouts  given  in  Appen¬ 
dices  T,  U,  V,  and  W. 


TABLE  3 

RESULTS  OF  THE  CURVILINEAR  REGRESSION  ANALYSIS 


Type  of  Curve 


Predictor 

linear 

quadratic 

cubic 

Variable  (X) 

2 

2 

2 

2 

2 

2 

r 

xy 

R 

s 

R 

s 

R 

s 

SRE 

-.069 

.005 

.002 

.007 

.002 

.072 

.002 

PRE 

-.009 

.000 

.002 

.043 

.002 

.047 

.002 

GRE 

.024 

.001 

.002 

.003 

.002 

.004 

.002 

WRE 

.242 

.058 

.002 

.058 

.002 

.062 

.002 

r  =  product  moment  correlation  between  the  predictor 

variable  X  and  the  criterion  variable  Y  (TP  score) 

2 

R  =  multiple  correlation  coefficient 
2 

s  =  residual  mean  square  (M.S.) 

2 

Consider  Table  3  and  the  evaluation  of  SRE  (X)  as  a  predictor  of  TP 
score  (Y).  The  product-  moment  correlation  between  SRE  and  TP  score 


Please  disregard  the  numerous  F-ratios  which  as  discussed  in  Appen¬ 
dix  S  are  of  little  value  for  this  data. 

2 

Note  that  because  the  results  of  Table  3  are  in  terms  of  centered 
data,  one  must  be  concerned  with  the  relative  magnitude  of  the  residual 
mean  square  from  attempting  to  fit  a  linear  model  to  fitting  a  cubic 
model  to  the  data. 
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is  -.069  (not  significantly  different  from  zero  at  the  a  =  .05  level). 
Therefore  SRE  seems  to  be  non-predictive  of  TP  score  (and  the  absloute 
level  of  technician  performance).  In  attempting  to  fit  a  linear, 
quadratic,  and  cubic  model  to  the  data  of  SRE  values  and  TP  scores,  the 
R2  values  were  .005,  .007,  and  .072  respectively.  However,  in  view  of 
the  fact  that  the  regression  mean  square (M.S.)  does  not  change  from  the 
linear  to  cubic  model,  it  would  be  just  as  well  to  chose  the  linear 
model  (particularly  since  R2  for  the  cubic  equation  is  only  slightly 
larger  than  .005).  Therefore,  the  best  regression  equation  is 

Y  =  5.458  -  0.447X. 


Because  X  and  Y_are  essentially  independent,  the  best  estimate  of  Y 
will  always  be  Y,  the  mean  of  the  observed  Y  values,  regradless  of  the 
observed_  SRE.  This  result  is  further  reflected  in  noticing  that  the 
sample  Y  is  5.284,  approximately  equal  to  5.458  -  the  Y-intercept  of 
the  regression  line.  Finally  one  must  conclude  that  the  SRE  is  a  very 
poor  estimator  of  TP  score  and  therefore  cannot  be  held  to  possess  signi¬ 
ficant  utility  for  widespread  implementation. 


Observing  the  results  of  Table  3  for  the  predictor  variables  PRE 
and  GRE,  one  will  have  to  draw  the  same  conclusions  as  the  above  on 
SRE.  In  fact  the  PRE  and  GRE  seem  to  be  even  poorer  estimators  of  TP  score 
than  SRE.  Only  the  WRE,  with  a  computer  printout  for  regression  given 
in  Appendix  W,  seems  to  possess  any  merit  as  an  estimator  of  TP  score. 

In  particular  the  product-moment  correlation  of  .242  does  offer  some  hope 
that  the  WRE  can  be  associated  with  on-the-job  performance.  Because  R2 
is  essentially  the  same  and  the  Residual  M.  S.  does  not  vary,  the  linear 
fit  would  seem  to  be  the  best  fit  to  the  sample  data.  The  linear  regres¬ 
sion  equation  is  given  by 


Y  =  3.66  +  2.646X 

where  Y  is  TP  score  and  X  is  WRE.  In  conclusion  it  must  be  said  that  WRE 
possesses  modest  value  as  an  estimator  of  absolute  technician  proficiency. 
Hopefully  further  research  into  this  estimator  and  associated  ones  will 
provide  a  more  valid  estimator  of  technician  performance  as  a  function 
of  magnitude  estimates  of  critical  incidents. 


SUMMARY  OF  RESULTS  AND  OBSERVATIONS 


The  data  collection  effort  conducted  by  NAVPERSRANDLAB  at  Destroyer 
Flotilla  NINE,  Pacific  Fleet,  involved  immediate  supervisors  evaluating 
technicians  in  the  eight  electronic  ratings  -  EM,  ET,  FT,  IC,  RD,  RM, 

ST,  and  TM.  From  the  evaluations  of  582  men  at  that  location  predictor 
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variable  information  was  collected  on  the  JPQ  ANSWER  SHEET  (Appendix  E) 
and  criterion  variable  information  was  collected  on  the  Technical  Pro¬ 
ficiency  Checkout  Form  (Appendix  C) .  The  following  conclusions  have 
been  made  in  this  paper  relative  to  four  estimators  of  technician  pro¬ 
ficiency  : 

a)  The  Series  Reliability  Estimate,  the  Parallel-Series 
Reliability  Estimate,  and  the  Geometric  Mean  Reliabil¬ 
ity  Estimate  (as  predictor  variables  derived  from  informa¬ 
tion  on  the  JPQ  ANSWER  SHEET)  do  not  seem  to  be  predictive 
of  the  absolute  level  of  technician  proficiency  as  deter- 
by  Technical  Proficiency  (TP)  Score  (the  criterion  vari¬ 
able  derived  from  information  on  the  Technical  Proficiency 
Checkout  Form).  In  every  case  the  product-moment  cor¬ 
relation  coefficient  could  not  be  termed  significantly 
different  from  zero  (at  the  a  =  .05  level).  This  con¬ 
clusion  was  made  in  view  of  the  results  of  a  curvilinear 
regression  analysis.  For  each  estimator  the  best  least 
squares  fit  to  the  sample  data  seemed  to  be  a  linear 

fit.  Finally  in  every  case  the  mean  of  the  sample 
criterion  variable  (technical  proficiency  score)  is  the 
best  estimate  of  an  absolute  level  of  technician  pro¬ 
ficiency.  As  such  none  of  these  predictor  variables 
seem  to  estimate  absolute  technician  proficiency  and 
cannot  at  this  stage  of  the  data  analysis  be  held  to 
possess  utility  for  widespread  implementation. 

b)  The  Weighted-Average  Reliability  Estimator  possesses 
moderate  validity  as  an  estimator  of  the  absolute  level 
of  technician  proficiency.  The  results  of  the  curvi¬ 
linear  regression  analysis  indicated  that  a  linear 
equation  was  the  best  fit  to  the  sample  data.  Further¬ 
more,  a  correlation  of  .242  with  the  criterion  variable 
indicates  that  it  possesses  a  possibility  for  future 
use  in  arriving  at  estimates  of  personnel  system 
performance  proficiency. 

The  above  results  were  made  under  the  assumption  that  the  Technical 
Proficiency  Checkout  Form  possesses  a  high  degree  of  validity  for  ap¬ 
praising  the  absolute  level  of  technician  proficiency.  This  assump¬ 
tion  is  made  here  as  it  was  by  Siegel  and  Federman  [12].  However, 
this  assumption  was  verified  for  technicians  in  avionic  ratings  by 
Siegel  and  Fischl  [13]. 
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Further  research  in  this  area  will  involve  a  comparison  of  the 
above  results  with  a  similar  statistical  analysis  applied  to  the  sample 
data  collected  at  Newport,  Rhode  Island,  and  Boston,  Mass.,  in  which  a 
total  of  367  men  were  involved.  Furthermore,  subsequent  data  analysis 
will  investigate  the  possibility  of  whether  or  not  any  of  the  predictor 
variables  can  be  validly  applied  to  particular  ratings,  rather  than  to 
the  population  of  eight  electronic  ratings  as  a  whole  as  this  report  had 
done.  Finally  the  possibility  of  combining  worthwhile  estimators  of 
electronic  ratings  performance  proficiency  in  a  meaningful  way  to  cal¬ 
culate  ship  or  squadron  efficiency  will  be  researched. 
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APPENDIX  A 


JOB  PERFORMANCE  QUESTIONNAIRE 

Name  of  Supervisor _  Rating _ Ship  or  Unit 


Instructions  to  Supervisor.  The  purpose  of  this  form  is  to  determine  the 
number  of  effective  and  ineffective  performances  you  have  observed  amongyour 
men  during  thepast  two  months.  We  are  only  interested  inthe  uncommonly  ef¬ 
fect  i  ue  and  the  unc  ommon  I  y  i  n e  f  f  e  c  t  i  u  e  performances. 

List  below  the  names  of  all  the  men  under  your  supervision  who  are  currently 
striking  for,  or  in  any  of  the  following  ratings:  DS,  EM,  ET,  FT,  IC,  MT,  RD,  RM, 
ST,  TD,  TM  (AE,  AT,  AQ,  AX).  If  you  supervise  more  than  one  of  these  ratings, 
please  use  a  separate  form  for  each  rating. 

Now,  considering  the  fleet  electronic  maintenance  objectives,  enter  your  estimate 
of  the  number  of  uncommonly  effective  (UE)  and  uncommonly  ineffective  (UI)  per¬ 
formances  during  the  past  two  months  for  each  man  being  rated.  Please  refer  to  the 
definitions  lists  for  the  meanings  of  the  JOB  ACTIVITIES  and  of  the  OBJECTIVES 
involved. 

The  first  line  has  been  filled  in  as  an  example.  The  supervisor  completing  the 
example  felt  that  Peter  Smith  had  ten  unusually  effective  performances  and  two 
unusually  ineffective  performances  while  performing  Electronic  Circuit  Anal- 
ses  when  considered  against  the  objectives  of  fleet  electronic  maintenance.  He 
also  felt  that  Smith  showed  two  uncommonly  effective  performances  in  the  area  of 
Electrosafety  and  four  uncommonly  ineffective  performances  in  Instruction. 

If  a  man  has  not  had  an  opportunity  to  perform  in  a  particular  area,  enter  a  dash(-); 
if  he  has  had  an  opportunity  but  has  not  shown  any  uncommonly  effective  or  ineffec¬ 
tive  performances,  enter  a  zero  (0). 


APPENDIX  B 


DESCRIPTION  OF  JOB  ACTIVITIES 

JOB  ACTIVITY  DESCRIPTION 


1.  Using  Reference  Materials — includes  the  following  type  of  activities: 

a.  use  of  supporting  reference’  materials 

b.  making  out  reports 

2.  Equipment  Operation — includes  the  following  type  of  activity: 


a.  operating  equipment,  electrical  and 
electronics  test  equipment,  and 
other  electronic  equipments 

3.  Electronic  Circuit  Analysis — includes  the  following  type  of  activities 

a.  understanding  the  principles  of 
electronic  circuitry 

b.  making  out  failure  reports 

c.  keeping  records  of  maintenance 
usage  data 

4.  Personnel  Relationships — includes  the  following  type  of  activity: 

a.  supervising  the  operation,  inspection, 
and  maintenance  of  electronic  equip¬ 
ments 

5.  Electro-safety — includes  the  following  type  of  activity: 

a.  using  safety  precautions  on  self  and 
equipment 

6.  Instruction — includes  the  following  type  of  activity: 

a.  teaching  others  how  to  inspect, 
operate,  and  maintain  electronic 
equipments 

7.  Electro-repair — includes  the  following  type  of  activity: 

a.  equipment  repair  in  the  shop 

8.  Electro-cognition — includes  the  following  type  of  activities: 

a.  maintenance  and  troubleshooting  of 
electronic  equipments 

b.  use  of  electronic  maintenance 
reference  materials 
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APPENDIX  C 


T  E  C  H  N  I  C  A  L  P  R  O  P  I  Cl  E  N-C-Y*  CHECK-OUT  FORM 

NAME  OF  SUPERVISOR _ RATING/RATE _ 


FULL  NAME  OF  MAN  EVALUATED 


SHIP  OR  UNIT _ LOCATION  _ DATE 


1. 


2. 


3. 


4. 


5. 


6. 

7. 


8. 


TASK  DESCRIPTION 


Capable  of  employing  safety  precautions  on  most 
of  this  unit's  equipment  with  which  his  rating 
is  concerned . 

Capable  of  replacing  most  of  this  unit's  equip¬ 
ment  with  which  his  rating  is  concerned.  . 

Capable  of  removing  most  of  this  unit's  equip¬ 
ment  with  which  his  rating  is  concerned. 

Capable  of  following  block  diagrams  for  most  of 
this  unit's  equipment  with  which  his  rating  is 
concerned . 

Capable  of  knowing  relationship  of  equipment  to 
other  related  equipment  with  which  his  rating 
is  concerned. 

Capable  of  calibrating  most  of  this  unit's  equip¬ 
ment  with  which  his  rating  is  concerned. 

Capable  of  trouble-shooting/isolated  mal¬ 
function^)  in  most  of  this  unit's  equipment 
with  which  his  rating  is  concerned* 

Capable  of  employing  electronic  principles 
involved  in*^naintenance  of  most  of  this  unit ' s 
equipment  with  which  his  rating  is  concerned. 


NOT 

CHECKED  CHECKED 


OUT 

OUT 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

MAKE  CERTAIN  THERE  IS  AN  "X"  IN  A  BOX  OPPOSITE  EACH  TASK  DESCRIPTION 


APPENDIX  D 


MEANINGS  OF  FLEET  ELECTRONICS  MAINTENANCE  OBJECTIVES 

1 .  Readiness 

To  maintain  efficiently  self,  subordinate  personnel,  equipment, 
and  systems  in  state  of  readiness  consistent  with  fleet  requirements. 

2.  Performance 


To  complete  any  given  mission  in  minimum  time  with  appropriate 
level  of  accuracy  and  reliability. 

3.  Operation 


To  obtain  optimum  system  output  when  equipment  is  operated,  i.e., 
output  characterized  by  precision  and  variability  appropriate  to  mission. 

4.  Safety 


To  carry  out  duties  with  maximum  protection  for  men  and  equip¬ 
ment  consistent  with  mission. 

5.  Preparation 


To  prepare  for  personnel  requirements  of  present  and  future  equip¬ 
ment,  systems,  and  situations  through  use  of  training  programs,  maintenance 
of  high  morale,  etc. 
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APPENDIX  E 


22 


APPENDIX  F 


REMINDER:  WHEN  TYPING  LAYOUT,  PLEASE  MAKE  SURE  THE  RIBBON  1$  IN  GOOD  CONDITION  SO  THAT 

YOU  WILL  HAVE  A  SHARP,  DARK  COPY  FOR  BETTER  REPRODUCTION. 
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APPENDIX  G 


The  purpose  of  this  section  is  to  examine  the  frequency  with  which 
the  two  cases : 

1.  a  technician  did  not  work  at  a  job  activity,  and 

2.  a  technician  received  ZUE  =  0  and  ZUI  =  0, 

occured  for  each  rating  and  job  activity  across  all  eleven  ships  participa¬ 
ting  in  the  project.  From  this  one  can  infer  on  the  extent  which  any  con¬ 
vention  for  estimating  performance  in  those  cases  would  effect  individual 
SRE,  PRE,  and  GRE  values. 

Refer  to  the  table  in  Appendix  H.  Each  square  in  the  table  represents 
the  number  and  proportion  of  technicians  by  rating  evaluated  at  San  Diego 
who  did  not  work  at  a  particular  job  activity  or  received  ZUE  =  0  and 
ZUI  =  0  by  their  supervisor.  Therefore,  on  job  activity  Number  1,  10  of 
the  EM's  (or  18.5%  of  the  EM's)  evaluated  at  San  Diego  either  do  not  work 
at  that  job  activity  (Using  Reference  Materials)  or  received  ZUE  =  0  and 
ZUI  =  0.  This,  however,  may  seem  a  tolerable  level  of  occurence  of  such 
cases,  but  when  the  proportion  of  such  cases  exceeds  .33,  one  should  begin 
to  consider  whether  the  performance  of  some  individuals  is  due  more  to  the 
convention  that  must  be  adopted  rather  than  to  the  individual’s  own  job 
effectiveness.  Of  the  64  squares  in  the  table,  46  squares  had  one- third 
or  more  of  the  men  in  some  rating  falling  into  the  two  cases  for  some  job 
activity,  25  squares  had  one-half  or  more  of  the  men  in  some  rating  falling 
into  the  two  cases,  and  most  critically,  6  squares  had  at  least  75%  of  the 
men  in  those  cases.  In  particular  the  RD  and  RM  ratings  were  particularly 
notorious  for  this  type  of  situation  occuring.  It  is  clear  in  the  RD  and 
RM  ratings  that  any  convention  adopted  will  probably  poorly  reflect  indivi¬ 
dual  performance  and  more  reflect  the  effect  of  the  convention.  No  rating 
seems  to  be  free  of  this  situation  for  some  job  activities,  however,  some 
ratings  demonstrate  this  effect  for  more  job  activities. 
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APPENDIX  H 


NUN3ER  AND  PROPORTION  OF  MEN  WHO  OID  NOT  WORK  AT  A  PARTICULAR  .JOR  ACTIVITY  OR  RECEIVED  SUE-0  AND  EUI-0  BY  RATING 

PATINO 


ACT. 

E>» 

ET 

FT 

IC 

no 

RM 

ST 

TM 

1 

10 

2* 

mm 

6 

31 

32 

23 

5 

0.  HS 

0.220 

0.143 

0.373 

0.372 

0.277 

0.179 

2 

1  1 

37 

34 

4 

29 

25 

21 

7 

n.  204 

0.314 

0.386 

o.oos 

0.340 

0.291 

0.253 

0.250 

3 

'  IB 

36 

41 

11 

78 

72 

39 

20 

0.333 

0,305 

0.466 

0.262 

0.940 

0.837 

0.470 

0.714 

4 

20 

60 

60 

20 

40 

50 

47 

8 

0. 370 

0.500 

0.682 

0.476 

0.590 

0.581 

0.566 

0  .286 

5 

1  1 

63 

54 

6 

47 

33 

35 

8 

0.204 

0.534 

0.614 

0.143 

0.566 

0.384 

0.422 

0.286 

27 

81 

60 

23 

62 

54 

— 

50 

1? 

0  .  5'JO 

0.666 

0.784 

0.548 

0.747 

0.628 

0,602 

0.429 

7 

9  , 

*36 

40 

3 

82 

81 

38 

18 

n.  167 

0.305 

0.557 

0.071 

0.088 

0.942 

0.458 

0.643 

8 

20 

41 

41 

8 

*2 

71 

30 

16 

n .  3  7  0 

0.34  7 

0.466 

0.100 

0 .9H** 

0.826 

0,361 

0.571 

NUMBER  OF  MEN  IN  EACH  RATING 

54  118  88 

42 

83 

86 

83 

28 
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APPENDIX  I 


This  section  will  be  concerned  with  justifying  the  hypothesis  that 
no  convention  can  be  adopted  per  ship  that  will  account  for  those  cases 
in  which  a  technician  either  does  not  work  at  a  particular  job  activity 
or  received  EUE  =  0  and  ZUI  =  0  from  his  supervisor.  As  an  example, 
observe  the  table  in  Appendix  J.  This  table  is  of  the  same  type  as 
that  previously  reported  on  for  all  men  at  San  Diego  given  in  Appendix 
G,  however,  it  is  reporting  on  only  one  typical  ship  out  of  the  eleven 
ships  in  the  project.  For  this  ship  there  were  8  (out  of  64)  instances 
where  those  two  cases  occured  for  all  men  in  some  job  activity  and  rating. 
The  other  ten  ships  demonstrated  16,  7,  8,  10,  13,  12,  5,  5,  5,  and  16 
(out  of  64)  such  instances.  Therefore,  it  is  impossible  to  form  an  aver¬ 
age  estimate  (or  some  composite  value)  per  ship  for  each  rating  and  job 
activity  based  upon  the  performance  of  individuals  in  that  rating  and 
job  activity  who  received  ZUE  ^  0  and/or  EUI  ^  0  if  in  fact  no  such 
individuals  exist. 
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APPENDIX  J 


NUMRER  AND  PROPORTION  OF  MEN  WHO  DID  HOT  WORK  AT  A  PART  I CULAR  JOB  ACTIVITY  OR  RECEIVED  :UE»0  AND  £UI»0 

BY  RATING  ON  A  PARTICULAR  SHIP  AT  SAN  DIEGO 

RATING 


ACT. 

EM 

FT 

FT 

1C 

RD 

RM 

ST 

TM 

1 

4 

0 

5 

0 

0 

9 

0 

0 

0.667 

0.310 

0.417 

0.0 

0.0 

0.900 

0.0 

0.0 

2 

8 

n 

7 

0 

0 

9 

0 

1 

• 

1.000 

0.444 

0.S83 

0.0 

0.0 

■  0 . 90  0 

0.0 

0.077 

.1 

5 

14 

7 

1 

0 

10 

0 

9 

0.A33 

0.483 

0.583 

0 . 1  b  7 

0.0 

l.ooo 

o.o 

0.692 

4 

6 

IB 

10 

3 

0 

9 

0 

s 

1  .000 

0.621 

0  .833 

o.soo 

0.0 

0.900 

0.0 

0. 3»5 

S 

6 

21 

8 

0 

0 

fl 

0 

1 

1  .000 

0.  724 

0.667 

0.0 

0.0 

0.400 

0.0 

0.077 

6 

5 

■  *21 

•  9 

0 

0 

10 

0 

5 

0.433 

0.724 

0.760 

0.0 

0.0 

1.000 

0.0 

0.345 

7 

5 

11 

12 

1 

0 

9 

0 

9 

0 ,033 

0.370 

1 .000 

0.147 

0.0 

0.900 

0.0 

0.692 

8 

8 

14 

■4 

1 

0 

10 

0 

6 

i  .ooo 

0  .482 

0  .  68 7 

0.167 

0.0 

1.000 

0.0 

0.462 

NUMBER  OF  MEN  IN  EACH  RATING 

6  20  12  60 


APPENDIX  K 


COMPOSITE  RELIABILITY  VALUES  FOR  TECHNICIANS  AT  SAN  DIEGO 


JOB 


ACTIVITY 

EM 

ET 

1 

0.8770 

0.6831 

2 

0.9050 

0.7733 

3 

0.8465 

0.6932 

4 

0.8639 

0.5987 

5 

0.9004 

0.7706 

6 

0.9333 

0.8481 

7 

0.8981 

0.7872 

8 

0.8744 

0.7097 

I 


RATING 


FT 

1C 

RD 

0. 7160 

0. 7466 

0.9257 

0.7802 

0.  8217 

0.9110 

0.7340 

0.7981 

0.9000 

0.7890 

0. 7692 

Ol 9  30  0 

0.9107 

0. 7865 

0.9012 

0.8435 

0. 8039 

0.9677 

0. 8701 

0. 8163 

1.0000 

0.7771 

0. 8105 

1.0000 

RM 

ST 

TM 

0.9310 

0.8537 

0.7333 

0.9497 

0. 8899 

0.8165 

1.0000 

0.8662 

0.8667 

0.9671 

0.8930 

0.7733 

0.9712 

0.9161 

0.7284 

0.9877 

0.8571 

0.  8974 

1.0000 

0.9078 

0.8269 

0.9949 

0.8571 

0.7419 
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APPENDIX  L 


The  purpose  of  this  section  is  to  demonstrate  the  greater  utility 
of  the  WRE  as  an  estimator  of  individual  performance.  Essentially  it 
is  a  better  type  of  estimator  in  that  it  is  not  dependent  on  a  conven¬ 
tion  to  be  adopted  for  the  case  whereby  a  man  did  not  work  at  a  particular 
job  activity.  As  such  the  convention  need  only  provide  reliability  ratios 
for  those  job  activities  for  which  the  man  being  evaluated  received  IUE  =  0 
and  IUI  =  0  from  his  supervisor. 

Consider  the  table  in  Appendix  M.  Each  square  in  the  table  represents 
a  breakdown  of  the  table  given  in  Appendix  G  into  the  number  (and  proportion) 
of  men  who  did  not  work  at  a  particular  job  activity  and  those  men  who 
received  IUE  =  0  and  EUI  =  0.  For  example,  on  job  activity  Number  1,  6 
(11%  of  the  EM's)  received  IUE  =  0  and  IUI  =  0  and  4  (7%  of  the  EM's)  did 
not  work  at  that  job  activity.  The  composite  reliability  values  need  then 
only  be  employed  on  11%  of  the  men  in  that  rating  and  job  activity  rather 
then  on  18.5%  of  the  men  as  required  by  the  SRE,  PRE,  and  GRE.  More  signi¬ 
ficantly,  in  the  case  of  RD’s  and  RM's  for  example,  at  most  52%  (as  compared 
to  98%  for  the  SRE,  PRE,  and  GRE)  of  the  men  in  those  ratings  derive 
reliability  ratios  for  some;  job  activities  from  the  composite  reliability 
table.  Clearly  this  is  a  significant  improvement  which  should  improve 
individual  performance  estimates.  The  statistical  analysis  in  the  main 
text  of  this  paper  verified  this  conjecture. 

Derivation  of  the  Weights  Employed  by  the  WRE 

On  the  JPQ  ANSWER  SHEET  in  Appendix  E  in  column  (c)  for  each  job 
activity  the  following  question  was  answered  by  the  supervisor  on  the  man 
he  is  evaluating: 

QUESTION  (c)  Considering  this  man’s  overall  performance,  it  is  your 

opinion  that  the  importance  of  this  job  activity,  as  a  factor 
in  determining  the  overall  performance  of  this  man,  is  best 
described  as  being: 

3.  of  central  and  primary  importance 
2.  a  significant  factor,  but  of  secondary  importance 
1.  of  only  moderate  importance  in  estimating  overall 
performance 

0.  of  little  or  no  importance 

The  weights  (w.)  for  the  it^1  job  activity  are  determined  by  the  formula: 
i  th 

If  the  supervisor  recorded  the  i  job  activity  as: 

of  central  and  primary  importance,  the  weight  w^  =  1.0 
of  secondary  importance,  the  weight  w^  =  .75 
of  moderate  importance,  the  weight  w_^  =  .5 
of  little  or  no  importance,  the  weight  w^  =  .25 
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APPENDIX  M 


> 


.4  NUMBER  AND  PROPORTION  OF  MF.N  WHO  RECEIVED  IUF.-0  AND  F.UI-0 

AND  NUMBER  AND  PROPORTION  OF  MEN  WHO  DIO  NOT  WORK  AT  A  PARTICULAR  JOB  ACTIVITY  BY  RATING 

PAT  I  NT, 


JOP  ACT. 

E* 

F.T 

FT 

[C 

RD 

RM 

ST 

TM 

\Z 

N9 

N/ 

N9 

NZ 

N9 

NZ 

N9 

N7  N9 

NZ 

N9 

NZ 

N9 

NZ 

N9 

t 

6 

4 

26 

0 

38 

'  5 

ft 

0 

28  3 

16 

16 

19 

4 

5 

0 

0.111 

0.0  74 

0.220 

0.0 

0.432 

0.057 

0.143 

0.0 

0.337  0.016 

0.186 

0. 186 

0.229 

0.048 

0.179 

0.0 

a 

5 

b 

37 

0 

33 

1 

4 

0 

29  0 

18 

21 

0 

7 

0 

0.043 

0.111 

0.314 

0.0 

0 . 3  7.5 

0.011 

0.095 

0.0 

0.349  0.0 

0.209 

0.081 

0.253 

0.0 

0.250 

0.0 

3 

1.3 

5 

31 

3 

37 

4 

11 

0 

4  3  .35 

35 

37 

29 

1  0 

17 

3 

0.?4l 

0.0V3 

0 , 2  (*  0 

0.025 

0.420 

0,0  45 

0.262 

0.0 

0.S18  0.422 

0.407 

0,430 

0.349 

0.120 

0.607 

0.107 

4 

IP 

8 

37 

23 

45 

15 

16 

4 

30  19 

20 

30 

36 

1  1 

7 

1 

0.222 

0.148 

0.314 

0.195 

0.51  l 

0.170 

0,381 

0.095 

0.361  0.229 

0.233 

0.349 

0.434 

0.133 

0.250 

0.036 

■3 

H 

3 

62 

1 

53 

1 

6 

0 

43  4 

26 

7 

34 

1 

7 

1 

0.  148 

0.056 

0.S25 

0 . 0  0  8 

0.602 

0.011  - 

0.143 

0.0 

0.518  0.048 

0.302 

0.081 

0.410 

0.017 

0.250 

0.036 

6 

18 

0 

5H 

23 

46 

24 

17 

b 

34  28 

22 

32 

29 

21 

9 

3 

0.333 

0.167 

0.49? 

0.195 

0.511 

0  .273 

0.405 

0.143 

0.410  0.337 

0.256 

0.372 

0.349 

0.263 

0.321 

0.107 

7 

4 

5 

3b 

1 

40 

9 

3 

0 

30  S2 

31 

50 

25 

13 

17 

1 

0.074 

O'.  043 

0  .?4  7 

0.008 

0.455 

0.102 

0.071 

0 . 0 

0.361  0.627 

0.360 

0.591 

0.301 

0.167 

0.607 

0.036 

fc 

1  1 

9 

41 

0 

37 

4 

8 

0 

32  50 

27 

44 

21 

9 

13 

3 

0.204 

0.167 

0.34  7 

0.0 

0.420 

0.045 

0  .  190 

0.0 

0.386  0.602 

.0.314 

0.512 

0.253 

0.108 

0.464 

0.107 

NUMBER  OF  MEN  IN  EACH  RATING 

54  118  88  4 2  83  86  83  28 

N:  °  HUMBER  AND  PROPORT  I  Of.’ OF  MEN  WHO  RECEIVEO  TUE-0  AMD  TUI-0  IN  THAT  JOB  ACTIVITY  ANO  RATING 
N3  =>  NUMBER  AND  PROPORTION  OF  MEN  WHO  OID  NOT  WORK  AT  THAT  JOR  ACTIVITY  IN  THAT  RATING 
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APPENDIX  N 


HISTOGRAM  OF  TP  SCORES  FOR  SAN  DIEGO  CAL 


FREQUENCY 

11 

37 

4  6 

51 

52 

81 

92 

68 

144 
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0000 

141 

0000 

138 

0000  ' 

135 
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132 
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129 
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126 
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««•« 

7? 
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54 
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51 
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42 
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39 
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36 
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«O0« 
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0000 
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33 
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27 
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12 
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0O00 

9 
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0000 

6 

000  4 
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3 
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6 

7 

a 

EACH  ****  EQUALS  3  POINTS 


APPENDIX  0 


HISTOGRAM  OF  SF.R1ES  RELIABILITY  ESTIMATES  (SRE)  FOR  TECHNICIANS  AT  SAN  0 1  EGO,  CALIFORNIA 


FREQUENCY  29  13  19  30  15  24  17  17  13  20  21  19  28  25  18  23  42  18  18  13  160 


159 

156 

153 

150 

147 

144 

141 

138 

135 

132 

129 
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120 
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99 

96 

93 

90 

87 

84 

81 

78 

75 

72 

69 

66 

63 

60 

57 

54 

51 

48 

45 

42 

39 

36 

33 

30 

27 

24 

21 

18 

15 

12 

9 

6 

3 


•  *  » 
**  » 
**  • 
*»  * 


•  • 
»  » 
*  * 
•  » 


*«* 

*** 

»•« 


CLASS 

INTERVAL 


10 


11  12  13  14  15 


16  17  18  19  20  21 


EACH  EQUALS  3  POINTS 


32 


APPENDIX  P 


HISTOGRAM  OF  SF.RI  ES-PARAI.LF.L  RELIABILITY  ESTIMATES  (PRE)  FOR  TECHNICIANS  AT  SAN  OIEGO,  CALIFORNIA 


FREQUENCY 
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40 
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35 
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*  *  * 

**# 

### 

#*• 

*•« 

#** 

30 

*#  * 

*  •  • 

*## 

•  *  * 

#*# 

*** 

27 

*  *  * 

*  •  • 

«## 

*** 
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APPENDIX  Q 


HISTOGRAM  OF  WRI GHTED-AVERAGE  RELIABILITY  ESTIMATES  (WRF)  FOR  TF.CHN  I C I  AMS  AT  SAN  OIEGO,  CAIIFORMIA 
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APPENDIX  R 


Class  Intervals  for  Histograms  of  Various  Reliability  Estimates 

CLASS  INTERVAL 


NUMBER 

CLASS  INTERVAL 

1 

[.96, 

1.0] 

2 

[.91, 

.96) 

3 

[.86, 

.91) 

4 

[-81, 

.86) 

5 

[-76, 

.81) 

6 

[.71, 

.76) 

7 

[.66, 

.71) 

8 

[.61, 

.  66) 

9 

[.56, 

.61) 

10 

[-51, 

.56) 

11 

[.46, 

.51) 

12 

[.41, 

.46) 

13 

[.36, 

.41) 

14 

[.31, 

.36) 

15 

[.26, 

.31) 

16 

[.21, 

.26) 

17 

[.16, 

.21) 

18 

[.11, 

.16) 

19 

[.06, 

.11) 

20 

[.01, 

.06) 

21 

[.oo. 

.01) 

The  class  interval  [a,  b)  is  defined  to  be  the  set  of  all  numbers 
x  such  that  a  <  x  <  b. 


35 


APPENDIX  S 


Let  Y  be  a  criterion  variable  and  X  the  predictor  variable.  If  N 
is  the  number  of  observations  on  each  of  X  and  Y,  then  =  [Y-|  ,  YN] 

is  the  row  vector  of  observations  on  Y.  For  a  given  matrix  A,  A'  will  be 
the  transpose  of  A.  In  particular  one  wishes  to  establish  a  linear  re¬ 
gression  equation  for  a  particular  response  Y  in  terms  of  the  variables 
X,  X2,  X^;  i.e.  it  is  desirable  to  establish  which  of  the  three  power 
curves,  linear,  quadratic,  or  cubic: 


Y  =  8  +  ?  8.X1 

o  !=]_  i 


p  =  1,  2,  or  3 


best  fit  the  data  obtained  on  X  and  Y.  In  matrix  notation  the  above 
equations,  in  terms  of  the  sample  observation  vectors,  can  be  expressed 
as : 


X  =  %.ft  +  E 

where  X  was  defined  above.  The  matrix  -X  =  [J,  X^,  X^]  where 

J*  =  [1,  ...,  IJ^xN*  =  ...»  X^]  for  i  =  1,  2,  or  3  and  therefore 

is  an  N  x  1  column  vector  of  observations.  ft'  =  [B0,  .  ..,  8^], 
the  vector  of  p  +  1  regression  parameters.  E*  =  [e^,  ...,  e^]  is  the 
vector  of  errors  due  to  lack  of  fit  in  the  particular  linear  model. 

One  wishes  to  estimate  ft  such  that  the  error  sum  of  squares  E'E  is 
minimized.  In  particular  a  least  squares  estimate  ft  of  ft  is  given  by 

ft  =  (X’X)-1*’* 


provided  the  square  matrix  X'X  is  nonsingular  and  the  regression  problem 
has  been  properly  expressed.  The  usual  assumption  one  makes  is  that  E*  is 
distributed  with  mean  [0,  ...,  0]^x^  and  variance-covariance  matrix  o2I 

where  I  is  the  identity  matrix.'  a ^  is  called  the  common  error  variance 
of  the  observations.  The  assumption  of  normality  of  the  error  vector  E 
is  not  required  in  order  to  obtain  the  least  square  estimates  for  any  of 
the  parameters  in  the  regression  equation.  Because  any  assumption  of 
normality  for  E  implies  that  the  observations  on  Y  or  X  are  normally 
distributed,  this  paper  will  be  concerned  with  only  least  squares  esti¬ 
mates.  One  cannot  discuss  normality  on  X  or  Y  because  of  prior  results 
(TABLE  2)  where  it  was  shown  that  the  distribution  of  TP  scores,  SRE,  PRE, 
and  GRE  are  not  normally  distributed.  Therefore,  it  is  imperative  that  the 
reader  be  aware  that  while  the  assumption  of  E  being  normally  distributed 
is  not  required  in  order  to  obtain  ft,  it  is  required  in  order  to  make 
tests  of  hypothesis,  as  contained  in  an  Analysis  of  Variance  Table, 
which  depend  on  the  assumption  of  normality.  These  tests  are  the  usual 
t-  or  F-tests  and  they  cannot  be  validly  applied  to  the  sample  data  ob¬ 
tained  at  San  Diego. 


36 


A  Least  Squares  Analysis  of  the  Sample  Data 


It  is  possible  that  a  least  squares  analysis  can  be  attempted  in¬ 
dependently  of  the  distributional  properties  of  the  criterion  and  pre¬ 
dictor  variables.  For  a  particular  predictor  variable  (X)  and  criterion 
variable  (Y),  the  square  of  the  multiple  correlation  coefficient  (R^)  pro¬ 
vides  a  measure  of  the  degree  to  which  a  particular  regression  model 
explains  variation  in  the  data.  is  defined  as 


Sum  of  squares  due  to  regression  -  Sum  of  squares  due  to  B0 


Total  (corrected)  sum  of  squares 


It  should  be  clear  that  the  larger  R  is,  the  better  the  fitted  equation 
explains  the  variation  in  the  data.  Furthermore,  0  <_  R^  1,  and  therefore 
R^  =  1  implies  a  perfect  fit.  However,  there  are  a  few  problems  with  this 
approach  (see,  for  example.  Draper  and  Smith  [5]  page  63).  Rather  one 
must  weight  the  value  of  R^  with  the  least  squares  estimate  (s^)  of  the 
common  error  variance  (a^)  where 


s  =  residual  mean  square 

=  (Y’Y  -  2J£)/(N  -  p  -  1) 


Of  course,  the  smaller  s  is  for  a  particular  model  under  consideration 
the  better  the  model  fits  the  data.  Therefore,  the  approach  is  to  weight 
increases  in  R^  with  decreases  in  s  in  order  to  arrive  at  the  best  least 
squares  model  for  the  data. 
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THE  PREDICTS*  VArIa.ILE(X>  IS  GAEa  AND  THE  C»  I TER  iPN  VAR  .1  ABLM  y )  IS  TP.  SC0RE  , 


T' 


0.926 

0,156 

0,882 

0.186 

.0,8*5 

0.21/ 

5.284  • 

2.329 

1.000 

o.vse 

0,697 

0,02* 

0.958 

1,000 

0,985 

0,008 

0,897 

o,9es 

1.000 

-0,001 

0.02*'' 

6,008  ■' 

-0,001 

1.  000 

TEST  MEANS  ANC  STANDARD  DEVIATI5NS 
t  X 

2  SODAS 
3'  X  CU9E 
4  Y 

C3RRELAT JON 

1  X 

2  S9UAS 

_3  _ * _ C  Cl  9  6 

"  4  Y 

FIRST,  5Eee*Ci  *ND  tM|«D  !:fcS«ct  PEL YN6KJ  ALS,  _ _ _  _ _ 

MULTIPLE  R  SQUARE  *  0 , c 01 

MULTIPLE  R  _*  _  0*02* _ 

N.D.F.l  •  1 

N  ,  D ,  F  , 2  »  580 

F  FOR  ANALTSJSJX.  VARIANCE  EN  «  • _ 0j326 _ 

BETA  weights 

0,02* 

.CONTRIBUTIONS  TO. mult  I.P.LE _C € 0 fe L XJ L£N _ _  .  _ 

AND  REGRESSION  FACTOR  LEAOINGS,  2ND  C6LUKM 
i  X  0,00l  i,000 

_S0UARED.JElJLJt6Jfitll5 _ 

0,001 
8  WEIGHTS 
0,353 


INTERCEPT  CONSTANT  «  4,95* 

. . . 

MULTIPLE  R  SCLARE  •  _ 0*503 _ 

MULTIPLE  R  *  0,058 

N.O.F.l  *  2 

N,c,r,2  «_  _  679 _ 

F  FOR  ANALYSTS  OF  VARIANCE  6N  R  *  0,96? 

BETA  WEIGHTS 
0,202  -0,186 


C0NTR I  BUT  JENS  TO  MULTIPLE  CO RcC. AT  I  EN 

AND  REGRESSION  FACTOR  LADINGS,  2N0  CBLUMN'I 
1  X  0 ,00  5 _ 6**08 _ 


SCUAR 


SQUARED  BETA  WEIGHTS 
_  0,0*1  _P*03* 

B  WEIGHTS 

3,003  *2,320 

INTERCEPT  CONSTANT  « 


>0,001 


0,130 


Jl5*S_ 


siiiiiiiiii::iai8iiij:i:is:itiBBiisiixi,»isiiiisii,iiiiiBa8Bx,,i,xiiaism 

MULTIPLE  R  SQUARE  »  0,00* 

-MULTIPLE. -B  JL _ 0.966 _ 

N, O.F.l  *  3 

N  , 0 ,F , 2  «  578 

F  F $R  ANALYSIS  OF  VARIANCE  ON  R  •  0,837 

BETA  WEIGHTS  . . 

0,729  *1,504  0,936 

CONTRIBUTIONS  TO  MULTIPLE  CBRRfcL AT  1  EN 

AND  REGRESSION  FACTOR  LOADINGS,  2ND  COLUMN) 

1  X  0,0i7  8,360 


JOLAR 
X  CLBE 


L0f01?_ 


*0,901 

SQUARED  BETA  wEIGMTS 

.  0,532  ..  2,510 _ 0*821 _ 

B  WEIGHTS 

10,859  *19,791  9,719 

_J H] ERCEPT  CONSTANT  »  *.*7« 


-0*1 15. 

*0,017 


anbva  table  fsr  pelynbmuls 

■iiiiixixiBiixBiiiiaiaxxiixiiiiiBxxxxiBiiBiB'iiiiaixxxiixiiiaiixixxixa 

RE  PICT  J  $N  r-LF  T0  LINEAR  FIT,  «ITw'l  CF  • . 0,001  “ . 

RESIDU*L  S.S,  *  0,999  rr  *  mo  RESIDUAL  m.s,  «  0,002 

F  F0R  LINEAR  FIT  • _  0,326 _ _ _ _ 

■xiitiBixxaiBxxiiixi|ixixxsixiixxiixaixiBisxii„iiBixixi:iixiiiiiiixxi 

REDUCTl  ON  DLF  T0JIENEHAL  CL' ATH  AT  J_C_£J  T_  W  J  Tw  QF_2,  • _ J,0O? _ 

reduction >'',s,"» . '  0,002  '  "  ’  . . . 

RESIDUAL  S.S,  *  0,997  PF-  «  579  RESIDUAL  M.S,  «  0,002 

F  FSR  OUACKATJC  FIT  f  0,982  _  _  . 

RFDUCTJSN  CUE  13  CLAORATJC  TfcRn  ALfcNfc,  WITH  1  OF,  »  0,003 

F  F0R  QUADRATIC  TERM  AL*nE  «  1,638 


REOUCTJ  BN  CUE  18  GENERAL  CUBIC  FIT  WITH  CF  3,  •  0,00* 

REDUCTION  . M.S,  * _ 0,001 . . . . . . 

RESIDUAL  S.S,  •  0,996  DF  •  576  RESIDUAL  H,S,  i  0.002 


F  FBR  GENERAL  CugiC  p I T  ■  R,83?_  _  _ 

REDUCTION  CUE  T 0  CUBIC  TERM  ALONE  wltw  I  OF  i  , 
F  FBR  CUBIC  TERM  aLSnE  *  0,5*8 


0,001 


AO 


. . . . . .  »«l  ■«»*****■**«.  :iii»j»iiiiii.i  «*■»  =  ««  ■nigmuisiiiin 

I  POLYNOMIAL  r  1  T T  I N r,  F0P  2  VAkIAPlCS-WRE  and  TP  SCGRP»  502  a0SF«VATlQN$, 


1 _ T.H£..?REOJC.T..PR_V.*fi  !>5l.P  (XL  IS  «RE«  * VD...t«6_-C.o I T.e.»  1  <S.M  VaRJaBLEIY)  15..TP  SCORE 


TEST  MEANS  AND  STANDARD  DEVIATIONS 

1  X  C.411  0,213 

2  SOUAR  _  P,Tt0 . . P.2AC 

3  X  COSE  0.3J5  i/,225 

4  T  5,20#  2,325 

.CORRELATUN  MATRIX . .  _  ...... 

1  X  1.300  0,963 

2  SQUAR  0.967  1.00C 

3  X_C09E  _  0,915 _ 0,904 

4  Y  0,2#2  0.231 


0,915 

0.9#6 

_ i.ooo, 

0,213 


FIRST,  SECONDHAND  ThIRD  DEGREE  POLYNOMIALS, 


MULTIPLE  R  SQUARE  »  .0,058 

MULTIPLE..R_« _ A  ui*2 _ _ 

N.D.F.i  ■  1 

N  ,  0  ,F  , 2  ■  560 

F  FOR  ANALYSIS  0F  VARIANCE  6N_H  •  _  _39,928 _ 

BETA  WEIGHTS 
0,242 

.CONTRIBUTIONS.  T.0_.MLLJ  JPUE-CORBELATtiN _ 

AND  REQRESSJ0N  FACTOR  LEADINGS,  2ND  COLUMN) 

1  *  3,058  1,000 

,souareo_beu_wejc.hj.s _ 

0,096 
B  WEIGHTS 

2,640  _ _ 

INTERCEPT  constant  «  3,067 


_  MULTIPLE  ..RJSCUARE.« _ 0^050 _ 

MULTIPLE  R  «  0,2«2 

N ,  D ,  F  ,  1  •  2 

_ N.D,F,2_f _ 579. _ 

F  F 0R  ANALYSIS  OF  VARIANCE  8N  R  •  17,902 

BETA  WEIGHTS 

i _  0,277..  .0,037 _ _ _ 

CONTRIBUTIONS  T9  MULTIPLE  CFRRel AT j £N 

AND  REGRESSION  FACTOR  LOADINGS,  2ND  COLUMN) 

_ _ ' _ 1 _  ...X _ a,o9? - 0."9_ - 

2  SCUAR  -9,009  0.9E6 

SQUARED  BETA  WEIGHTS 

. . 0,077..  „.0,..001 _ 

B  HEIGHTS 

3,038  rO ,396 

_ I  N  T  E  R  C  E.P.T_C  ON  ST  ANt  « _ 3  ,  .9  7  7 _ 


MULTIPLE  R  SQUARE  «  0,002 

L _ B  UL  T  LP.L  E_R-.« _ 0  j  l  >J _ _ 

N.D.F.l  «  3 

N,D,F.,2  «  974 

F  F0R  ANALYSIS'  OF  VARIANCE  0N  R  •  12,939 

BETA  HEIGHTS 
•0,296  1,302  -0,914 

CON  T9 1  BUT  I ENS  TO  MULTIPLE  CER°ELATI6N 

AND  REGRESSION  MCT0R  LEADINGS,  2ND  COLUMN) 

1  X  *0,002  0,960 

_ 2 S  0  U  A.R _ .0  ,.3l? _ 0X9  2.4 _ 

3  X  CUBE  -0,194  0,891 

SQUARED  BETA  WEIGHTS 

_  0,067  _ 1,90? _ 0,015 _ : _ 

3  HEIGHTS 

•2,629  13,380  -6,888 

_  JNTERC6PJL  constant  ■ _ 4,J24_  _ _ ... 


an9v a  table  fbr  polynomials 


REDUCT  I  BN  DIETS  LINEAR  FIT,  WITH  1  DF  «  0,05* 

RESIDUAL  S,S,  «  0,942  DF  *  5*0  RESIDUAL  M,S,  *  0,002 

F  F PR  LINEAR  FIT  «  .  35,928 _ _  _ _ _ _ - . 


REDUCT  I  CN  CUE  TO  GENERAL  GUADR  A_T_I  C..F  I  T_  H.I  TM  _DF  .2 _ Qjfl56 _ 

REDUCT 1 6N  M.S.".  ‘  ‘  6,029 

RESIDUAL  S.S,  «  0,942  Df  »  579  RESIDUAL  H,S,  *  0,003 

F  FPR  QUADRATIC  FIT  |  17,902  ..  .  .  _ _ _ _ _ _ 

REDUCTION  CUE  TO  CLAnHATjc  TERM  ALONE,  with  1  DF,  •  0,000 

F  FOR  QUADRATIC  TERM  ALONP  *  0,055 


REDUCES  CUE  T6  GENERAL  CLRIC  FIT  WITH  Er  S,  »  0,003 

RFDUCTI0N  M.S,_»  .  .9,021 . .  ..  .  . .  -  - 

RESIDUAL  S , $ ,  *  •  0  i V 3 0  DF  .  578  RESIDUAL  M,S,  • 
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13.  ABSTRACT 

The  purpose  of  this  research  effort  is  to  validate  the  utility  and  effectiveness 
of  a  unique  human  performance  measurement  technique  developed  under  ONR  contract 
(N0001467C0107) .  Performance  data  on  eight  Navy  ratings  was  collected  from 
ships  of  LANTFLT  and  PACFLT.  This  report  is  the  first  in  a  series  of  technical 
reports  on  the  statistical  analysis  of  that  data.  In  particular,  a  statistical 
analysis  is  provided  on  performance  related  data  for  electronic  maintenance 
personnel  sampled  from  11  ships  of  CRUDESFLOT  NINE,  San  Diego,  California.  Four 
different  performance  estimators,  as  functions  of  critical  incidents,  are  evaluated 
with  respect  to  a  performance  criteria.  A  detailed  explanation  of  the  distribu¬ 
tional  properties  of  the  performance  estimators  is  presented  and  an  explanation 
of  the  factors  that  lead  to  the  adoption  of  a  curvilinear  regression  analysis  for 
analysis  of  the  data  is  discussed. 

The  results  of  the  statistical  analysis  indicated  that  a  certain  combination 
of  the  performance  data  possessed  moderate  validity  for  measuring  the  absolute 
level  of  technician  performance.  Detailed  analysis  of  the  performance  data 
also  identified  the  areas  of  difficulty  that  have  to  be  avoided  in  order  to 
improve  upon  the  validity  of  each  of  the  performance  estimators.  These  and 
other  preliminary  results  indicate  that  the  technique  possesses  merit  for  further 
development  and  research.  The  Naval  Personnel  Research  and  Development  Laboratory 
is  continuing  research  on  this  technique  with  respect  to  the  totality  of  data 
collected . 
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